Testing Task Force Quarterly Update

This article originally appeared in The Bar Examiner print edition, Fall 2020 (Vol. 89, No. 1), pp. 93–94.

For the latest Testing Task Force news—and to sign up to receive updates with the latest Task Force news, research, and blog posts—visit testingtaskforce.org.As this update is being written, the COVID-19 outbreak continues to impact the work of everyone in the country, and although the Testing Task Force has had to make adjustments, we remain on track and are entering the “home stretch” of our three-phase study to design the next generation of the bar examination. At the beginning of March 2020, the Task Force published a report on the results of our Phase 2 nationwide practice analysis. At that time, the coronavirus pandemic had started to unfold in the United States, so the Phase 2 report flew under the radar for many because of more pressing world concerns. However, the report is worth taking the time to read as the results provide a reliable and robust set of data on the work performed by newly licensed lawyers and the knowledge, skills, and abilities they need to practice—which is essential information for determining competencies to be assessed on the bar exam.

The Phase 2 report was followed by significant Phase 3 work undertaken this summer. The Task Force convened two committees that met by video conference to develop recommendations for the blueprint and design of the new examination based on the results of the practice analysis and the opinions and ideas voiced by stakeholders during Phase 1, as well as the professional judgment and experience of the committee members. The blueprint committee, composed of 17 newly licensed and experienced lawyers (including 14 women and 11 people of color) from 13 jurisdictions who work in a variety of practice areas and settings, identified the knowledge, skills, and abilities that should be assessed on the next generation of the bar exam. None of the blueprint committee members were affiliated with the bar exam. The test design committee, composed of 28 bar administrators, bar examiners, justices, and legal educators and deans (including 10 women and 7 people of color), focused on methods of assessment, the timing and sequencing of those assessments, procedures for scoring, and other important features of test design and delivery.

As the Task Force works to complete the final phase of our study, it is exciting to see preliminary recommendations forming and a vision for the next bar exam starting to emerge. Consistent with the Phase 1 and Phase 2 data, both the blueprint and test design committees endorsed testing fewer subject areas. The blueprint committee members, all of whom are practicing lawyers, tended to mostly be unanimous in their opinions on what content should be included on the bar exam even though their areas of practice and practice settings are richly varied; the test design committee members were mostly unanimous that the content assessed should be limited to core legal subjects and greater emphasis should be given to assessing lawyering skills, which echoes what stakeholders said during Phase 1. The subjects and skills identified for assessment are set out in Table 1. Note: both committees also supported greater emphasis on assessing Professional Responsibility and Ethics but in a way that avoids redundancy with the current coverage on the Multistate Professional Responsibility Examination, which will likely remain a stand-alone exam. Additionally, members of the test design committee expressed concern that assessment of some skills, such as those related to Client Counseling and Advising, Client Relationship and Management, and Negotiation and Dispute Resolution, could introduce bias and subjectivity in grading, so care will be taken in considering whether and how to measure such skills.

Table 1: Subjects and skills being considered for assessment on the next generation of the bar exam

In comparison to the blueprint committee, the test design committee, whose members occupy various roles in educating and licensing lawyers, tended to be less unanimous in their opinions on test design issues. That is not to suggest that their opinions always fell along lines of educators versus bar examiners. Rather, their discussion rightly reflected the interconnectedness and complexity of some of the test design issues and provided valuable insight into the benefits and challenges of various approaches to those design issues. The results of these two committee meetings were reviewed by NCBE’s Technical Advisory Panel of independent psychometric experts to provide additional measurement expertise, ideas, and perspectives to those of the Task Force’s independent research consultant and NCBE’s own measurement staff. The Task Force is evaluating the input from both committees and our psychometric experts to refine the combined output into preliminary recommendations before deciding upon a final set of recommendations to submit to NCBE’s Board of Trustees.

Both the Phase 1 and Phase 2 reports are available on the Task Force’s website: testingtaskforce.org/research. We plan to post our Phase 3 report covering the work done by the blueprint and test design committees in November. The Task Force’s recommendations and comprehensive report will be submitted to the NCBE Board of Trustees in January and will be made public in early 2021. Please subscribe to the website to stay informed as we complete our study.

Contact us to request a pdf file of the original article as it appeared in the print edition.